TeekoTeacher: A Tool for Learning Good Teeko Strategies 


Mihailo Despotovi¢' and Vladimir Srdanovié 


'Silicon Genetics 
Redwood City, California, USA 
E-mail: mihailod@hotmail.com or mihailod@silicongenetics.com 


*Center For Multidisciplinary Studies 
University of Belgrade, Belgrade, Serbia and Montenegro 
E-mail: vladats@eunet.yu 


Abstract 


Teeko is a two-player zero-sum game with perfect information. The game tree complexity and the 
estimated state space complexity place it roughly in the same class with games such as Connect-4, Awari 
and Othello. This paper presents an educational tool (‘TeekoTeacher’) capable of teaching good Teeko 
strategies to human players. TeekoTeacher uses a database of moves created by another application 
(‘TeekoPlayer’, a Teeko playing program.) TeekoTeacher is an interactive GUI-driven and convenient 
web based (Java applet) tool that allows human players to play against themselves and to learn proposed 
moves. The paper also discusses the relevance of the tools and methods used to address other issues of 
E-Learning. 


Introduction 


Distance learning is an educational paradigm in which the instructor and students are geographically 
remote from each other. 


E-learning (also called ‘online learning’) is a distance learning technique that uses the Internet as the 
medium for delivering the course materials and teacher-student interaction. 


Benefits of E-learning (NIIT, 2003) include flexibility, convenience, accessibility, inexpensive worldwide 
distribution, travel cost/time savings, and ease of updating course materials. Issues relating to E-learning 
include the bandwidth limitations of the communication channel, and the resources (including time and 
money) invested in research, development, deployment, and maintenance of the tools. 


In a typical E-learning scenario, students would use their computers to connect to the content providers 
(online classrooms.) In addition to standard text content, these online classrooms usually utilize various 
multimedia tools to enhance the overall learning experience. The tools are divided (ClassesUSA, 2003) 
into: synchronous (require teacher’s real-time participation and interaction; examples: chat rooms, 
interactive TV and audio/video conferencing) and asynchronous (don’t require teacher’s real-time 
participation; examples: email, message boards, audio/video streams and interactive computer games.) E- 
learning implementations usually contain both types of tools. 


One example of an asynchronous e-learning tool is an interactive computer game. In general, in addition 
to the previously mentioned benefits, games provide an entertaining interaction during which the learning 
material is delivered implicitly. Strategic games are of special interest to E-learning applications because, 


by their nature, they don’t require big bandwidth and short network latency. Another interesting and 
important aspect of strategic games is that good strategies can be learned both from the game experts and 
during autonomous play. 


The general idea behind this paper follows from the previous paragraph: first, use machine learning to 
learn and then use E-learning to deliver the learned content. In this case, the content is a set of good game 
strategies. We present this ‘learning cycle’ using a simple strategic game Teeko. First, we present the 
game and its rules. Then, we describe a program TeekoPlayer that plays the game and learns how to play 
better. Finally, we present an interactive tool TeekoTeacher that takes advantage of the knowledge 
generated by TeekoPlayer and teach humans the learned good strategies extracted from that knowledge. 


About Teeko 


The game Teeko was invented in the middle of the 20" century and the game gained some popularity 
during the 1950’s (Scarne, 1955.) Scarne stated that the game is as playable as some older and more 
established board games like Checkers, Chess and Go. He was predicting a very bright future for the 
game and, according to him, some very popular celebrities already played it (Humphrey Bogart and 
Marilyn Monroe among others.) The game was also somewhat popular in the beginning of the Internet 
era because it was quite convenient to play it over the emerging email system. 


Using the terminology from Game Theory, Teeko can be classified as a two-player zero-sum 
deterministic sudden-death strategic game with perfect information and perfect memory. The rules of so- 
called ‘standard’ Teeko are relatively simple. The game is played on a 5x5 chess-like board. Each player 
has four equal pieces. The first four moves define the opening phase and usually no one wins at this point 
(although some trivial wins are theoretically possible.) During this phase, each player alternatively puts 
one piece on an unoccupied field on the board. The pieces are then to be moved only to unoccupied 
adjacent fields. The goal of the game is to make a four in a row, column, diagonal or 2x2 square. It’s 
easy to calculate that the total number of winning positions is 44. 


At the end of 1998, Teeko has been solved using a brute force method (Steele, 1998.) The solution was ‘a 
draw.’ This means that both players can guarantee not losing the game. The solving algorithm required 
about 200 MB of RAM and about 32 hours of computing time on a machine with a 300MHz Sun 
UltraSPARC CPU. 


Learning Teeko: TeekoPlayer Computer Program 


The first encounter with Teeko the authors had, was during creation of the “TeekoPlayer’ program 
(Despotovié, 2001.) The high-level research goal was to revisit an important artificial intelligence debate 
known as ‘search versus knowledge.’ Namely, we wanted to try to apply a combination of heuristic, 
searching and learning (with the emphasis on learning) to an algorithmic game theory problem and, in 
particular, to a specific game. We found Teeko to be an excellent test case for our experiments and 
applications of the theoretical results. 


TeekoPlayer was developed and used to examine the performance of the proposed learning algorithm. 
TeekoPlayer is a C program capable of playing, and learning to play better, the game of Teeko. The main 


features of TeekoPlayer are: 


¢ Plays the game of Teeko 


¢ Can play as the first or the second player 

¢ Has a ‘show/hide thinking’ option 

¢ Has the ability to learn in order to improve its game 

¢ Has on/off switches for heuristics, searching and learning modules 

¢ Has ‘self-play’ (the ability to play against itself) 

¢ Has ‘batch-play’ (the ability to play and record games without human intervention) 
¢ Keeps track of all games and makes convenient statistical data 


The main knowledge representation problems solved by TeekoPlayer are: 


¢ Storing and retrieving game positions 
¢ Storing and manipulating data generated by the learning algorithm 
* Generating all legal moves from a given position 


TeekoPlayer uses internal and external knowledge representation for game positions. Internally, 
TeekoPlayer handles all positions as 5x5 matrices. Externally, they are represented as chunks of a flat 
file (the ‘long-term memory.’) This file consists of one or more ‘memory elements’ each describing a 
position together with the number of games won and lost in which that position was a part of the strategy 
employed (we will refer to these numbers as winning and losing attributes of the position.) The 
fundamental idea was that TeekoPlayer should learn by adjusting these attributes during the course of 
many games. 


The game-playing engine consists of three main parts: 


¢ Heuristics module 
¢ Searching module 
¢ Learning module 


Heuristics module proposes heuristic moves such as ‘make four in a row’ or ‘make threatening three in a 
row.’ During the opening phase, only heuristic moves are used. They were obtained by formalizing the 
moves of human experts. Later, during the game, the heuristics module will be consulted for each 
position and, if it’s possible, this module will respond with a quick and efficient move. 


Searching module implements a simple one-move (ply and reply) look-ahead minimax algorithm. This 
algorithm examines the next two positions. 


Learning module implements the learning algorithm. The learning module uses ideas based on 
reinforcement learning (Sutton and Barto, 2000) and it takes advantage of the experience accumulated 
during the number of games played. This technique is also known as experience based learning (Jong 
and Schultz, 1988.) 


The learning algorithm is based on the following three concepts: 


¢ Storing. During the game, the learning module will be saving all visited positions into the short-term 
memory file. 

* Updating. After the game is finished, the learning algorithm will change the long-term memory 
according to the outcome of the game. If the game was a draw, no action will be taken. Otherwise, a 
lookup for each recorded position in the short-term memory file will be conducted in the long-term 
memory file. If the position cannot be found, it will be appended to the long-term memory file. 


Then, if the outcome of the game was won for the first player, the winning attribute will be increased 
by one; otherwise the losing attribute will be increased by one. 

¢ Recalling. Takes place during the game and its role is to recommend the position with the highest 
winning score reachable within one move from the current position. 


The final move is always decided by comparing the moves recommended by the heuristic module, the 
search module, and the learning module (if all of them are enabled.) If the learning module had a move to 
recommend, that move would be played. In this manner, we emphasize the impact of the learning 
algorithm to the overall style of play. In the general case, the learning module would not always have a 
move to recommend (or, it might have been simply disabled.) If this happens, TeekoPlayer will compare 
moves recommended by the heuristic and search modules (if enabled) and play the better one (in terms of 
the evaluation function used for the minimax search.) If there is no heuristic move to compare to, 
TeekoPlayer will play the move generated by the search module. 


The move selection process described above was very practical since all three modules could be 
independently turned on and off and provide a nice sandbox for conducting various ‘search vs. learning’ 
or ‘heuristics vs. search’ and other experiments. 
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Picture 1: TeekoPlayer’s interactive and batch mode 


In the interactive mode (left of Picture 1), TeekoPlayer displays the board and the thinking process. The 
most important information here is the eval value of the next move. The bigger this value is the better the 
next move is. In the batch mode (right of Picture 1), TeekoPlayer will just display the current game 
number, game outcome and the total number of moves played. 


Teaching Teeko: TeekoTeacher Educational Tool 


After the work on the TeekoPlayer was done, we realized that the long-term memory file created as a side 
effect of the learning process could be potentially used for some other interesting purposes. Namely, we 
can consider this file as a database of ‘annotated’ Teeko positions. Mining this database could tell us 
how good a given position is in the context of the database and the learning algorithm. However, in order 
to implement this idea we have to define what we mean by ‘good in the terms of the database and the 
learning algorithm.’ 


For example, it’s easy to observe that if a position’s winning attribute equals 8 and losing attribute equals 
2, we can claim that this position was part of a winning strategy (for the corresponding player) in 80% of 
the games that utilized it. We will refer to this percentage as winning percentage. But knowing the 
winning percentage of a single position alone is not sufficient. In general, there would be many other 
positions with both better and worse winning percentages. Therefore, we have to find a way to actually 
compare the winning percentage of a given position to all other interesting positions. Let’s now define 


‘interesting.’ If we have a particular Teeko position and if we know which player’s move it is, we can 
easily generate all successors of that position. It’s natural to use the successors as interesting positions 
because they represent the beginnings of all valid continuation strategies. 


Now we can define the complete procedure of suggesting the next position from a given position: 


1. Initially, a position X is given 

2. Generate all successors: X —+{Xx, GIN cus Xx, }= S 

3. Restrict S by mining the database: s—{x,, Kaveiey X,, }= Spx GS 

4. Calculate the winning percentages for all positions in Sp, : Spp —+{w, y Woy sees w, } 

5. Find the maximum winning percentage: {w, y Woy eeey Wn \__, Wax = max{w, eWay ces w,,} 


i=l,m 


6. Recommend the position with the maximum winning percentage: S,,—> X, 


Wouldn’t it be possible then to use the database and the described procedure to actually reverse the whole 
learning process and use it to teach human players how to play better? In order to try to answer to this 
intriguing question, we created an educational tool (TeekoTeacher) capable of mining the database and 
presenting the mined information to human players. 


The main features of TeekoTeacher are: 


¢ Parsing the database of positions created by TeekoPlayer 
¢ Heuristics for the opening phase of the game 

¢ Generating the valid successors of a given position 

¢ Searching for a given position in the database 

¢ Deciding which position to suggest to human players 

¢ Allowing two players to play against each other 

¢ Advising the first, the second, or both, players 

¢ Logging facilities 


Here is the global algorithm of TeekoTeacher (in pseudo language): 


while(opening() ) 
suggestOpeningMove () ; 


while(!endOfGame () ) 
{ 
successors = findAllSuccessors(); 
dbSuccessors = findSuccessorsInDatabase(sucessors) ; 
if (dbSuccessors.size() > 0) 
{ 
best = findTheBestSuccessor(dbSuccessors) ; 
recommendMove (best) ; 
} 
} 


The algorithm takes over after the 4" reply. During the opening phase (first four moves), TeekoPlayer 
uses recursive heuristics. Therefore, the natural solution to teach the opening was to simply suggest the 
very same moves in TeekoTeacher. This means that during the opening phase, TeekoTeacher is 


suggesting pure heuristic moves. The rules that define those heuristic moves are presented in the 
following table (we used the board enumeration convention shown in Picture 2): 


The only non trivial implementation issue here was that we had to take care of situations where human 
players refused to play one or more of the suggested opening moves. Fortunately, the presented heuristics 
still provided a reasonable robustness even in those situations. 
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[Java Applet Window 
Picture 2: TeekoTeacher Applet is suggesting a move 


We implemented TeekoTeacher as a Java applet (Picture 2.) It can be easily deployed to various clients 
using an almost trivial HTML page and delivered over the HTTP protocol. The database of positions 
generated by TeekoPlayer is an integral part of the applet. During the game, if TeekoTeacher encounters 
a move that can be proposed, it will present that proposal to the players by emphasizing the appropriate 
fields on the board and logging the appropriate information. 


Further Relations to E-Learning 


Here we are going to explain how TeekoTeacher provides E-learning benefits and addresses E-learning 
issues. 


Being a Java applet, TeekoTeacher automatically brings inexpensive distribution, reduced travel, 
accessibility and convenience. After the applet is deployed on a server, any Java-capable browser can be 
used to access the tool. ‘Ease of updating course materials’ translates to ‘updating the database of 
strategies.’ Currently, the database is a part of the applet. Every time the database has to be updated, we 
simply replace the applet file on the server. The very next time a user tries to load the applet, she will 
download the new version (technically, for this to happen automatically, the browser has to be set to not 
cache applets which is normally the default behavior.) Another approach would be to have the database 
as a separate file with a pointer (URL) and to let the user (or program) chooses which database to use. An 
advantage of that scenario is that the program and database changes are completely independent. 


One of the main issues of E-learning is the bandwidth limitation of the communication channel. If the 
tool is an applet, the issue shows only as the initial hit to the network occurring when browsers are 
downloading the applet. In the case of TeekoTeacher, this is the only networking operation and after the 
download is complete, no more network operations take place. One way to give the appearance of 
increased bandwidth is to use compression on the source and decompression on the destination. Java 
allows this by supporting compressed applets. We used the default Java compression tool (jar) to make 
the TeekoTeacher applet file. TeekoTeacher is a small application and the download of the compressed 
application and database is comparable to the downloading of one average jpeg picture (sub 100KB.) It’s 
doubtful that the size of TeekoTeacher itself will ever be an issue in this context. However, the database 
might grow in the future and there are many techniques that can be implemented to improve the download 
speed (one of them, for example, is to have the database outside the program and then apply incremental 
updates.) 


Another issue of E-learning is the amount of resources invested in development, deployment and 
maintenance of the tools. TeekoPlayer and TeekoTeacher were both developed from scratch and for 
academic purposes rather than as fully-fledged commercial applications. This is especially true for 
TeekoTeacher, which was developed in a relatively short period of time, and as a relatively simple proof 
of concept. Therefore, we cannot further address this issue here except to mention that we used 
standardized, portable and mostly free tools that we were proficient with (C, Java, HTML, and HTTP.) 


Concluding Remarks 


We presented one simple implementation of a potentially powerful idea: use machine learning to learn the 
content and then use E-learning to deliver the content. To prove the validity of this idea we used a simple 
strategic game. First, we learned some strategies and then we delivered that knowledge. The delivery 
technologies we used were standard, portable and easily deployable. 


We believe that this, or a similar approach, can be employed in many fields where machine learning is 
capable of acquiring a significant amount of domain knowledge, and that learning algorithms (with help 


from human experts) combined with attractive game-like interactive tools, may produce some very 
sophisticated virtual teachers. 
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